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Abstract 

Objectives To apply a programme logic model to evaluate the effectiveness of a new therapy service 
for children with special needs who were in transition from pre-school to kindergarten. 

Setting A children's outpatient rehabilitation centre in Ontario, Canada. 

Main outcomes The short-term outcomes included parents' perceptions of the transition process 
itself and the information they required, the children's skill development for the transition to 
kindergarten, and parents' perceptions of services and satisfaction with resources. 

Methods A combination of quantitative methods [Goal Attainment Scaling (GAS), Measure of 
Processes of Care (MPOC), Client Satisfaction Questionnaire (CSQ)] and qualitative interviews were 
used to evaluate both the process (‘Outputs') and outcomes (' Short-term objectives' ) of the new 
therapy service. 

Results The children involved in the evaluation met or exceeded goals that were set by therapists 
and parents. Parents' perceptions of, and satisfaction with, the new service were higher than the 
provincial average. Qualitative data from interviews with parents and service providers supported 
the findings from standardized measures, and provided suggestions for future service delivery. 
Conclusions The programme logic model provided researchers and service providers a collaborative 
and systematic approach to conducting programme evaluation in a relatively short-time frame. It 
appears to be a useful option for evaluation of other children's services. 
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Introduction 

In an effort to bring evidence into practice, many 
agencies and services have identified programme 
evaluation as an essential part of health care. Ser- 
vice providers and managers recognize the impor- 
tance of evaluating their programmes to ensure 
that they are effective. Unfortunately, many who 
work in child health are unsure where to start in 
evaluating their programmes. Numerous text- 


books and models are available, but they may 
appear to be overwhelming to health care providers 
who do not have any training or experience in this 
area. 

This article describes an innovative, easy-to-use 
approach to programme evaluation that involves 
the use of a programme logic model (Rush & 
Ogborne 1991). An application of this approach in 
a children’s outpatient rehabilitation centre is 
presented. 
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Background information 

Programme evaluation 

A programme evaluation is a systematic applica- 
tion of research procedures that investigates the 
impact and effectiveness of a programme/service 
(Favaro & Ferris 1988; Taylor & Botschner 1998; 
Letts et al. 1999). A programme is defined as ‘an 
organized set of activities that are managed 
towards a particular set of goals . . .’ (Rutman 1980, 
p. 42). 

The evaluation of programmes has been in exist- 
ence for many years in all sectors of human ser- 
vices, including health care. Programme evaluation 
is recognized as a type of inquiry that is different 
from basic academic research (Patton 1986). It is 
viewed as useful, or ‘applied’ research that informs 
decision making and promotes action being taken 
to make services more effective (Patton 1986, 
1990). Programme evaluation has emerged as a 
unique professional field of practice, with associa- 
tions, journals and guiding principles directed 
towards evaluators (Newman et al. 1995). 

There is a broad range of frameworks and 
methodologies available to evaluate programmes. 
Experts agree that the choice depends on multiple 
factors such as the purpose of the evaluation, the 
information needs, resources and values of the 
stakeholders, and the desired outcomes (Patton 
1990; Greene 2000; Eccles et al. 2003). Recent liter- 
ature has supported the benefits of involving 
stakeholders directly in the evaluation process to 
contribute to the effectiveness of programmes 
(Population Elealth Directorate 1996; Centers for 
Disease Control and Prevention 1999; Gilliam et al. 
2002; Zukoski 8t Luluquisen 2002). 

Programme evaluation is recognized as an 
essential part of evidence-based practice in today’s 
health care climate (Letts et al. 1999). It is also con- 
nected to quality assurance and improvement 
practices (Ovretveit 8t Gustafson 2002). Health 
care literature provides numerous examples from 
around the world of the application of programme 
evaluation in public health, health promotion and 
disease prevention (e.g. Centers for Disease 
Control and Prevention; Goodstadt et al. 2001) 
and national or state/regional programmes (Evans 


et al. 2001; Leese et al. 2001; Pirkis et al. 2001; 
Stoltzfus 8t Pillai 2002). Most of the literature on 
programme evaluation at a local, individual service 
level focuses on the effectiveness of the service 
itself, and does not emphasize the framework or 
methods used for the evaluation. This article places 
emphasis on the description and application of a 
specific evaluation model in health care, in an 
effort to increase service providers’ and adminis- 
trators’ awareness of a useful approach to evaluate 
their programmes. 

The programme logic model 

The programme logic model (Rush 8t Ogborne 
1991) involves a systematic approach that is appli- 
cable to most settings and programmes, and has 
been used successfully in a variety of settings (Rush 
& Ogborne 1991; Letts et al. 1999). Users have 
found it to be clear and helpful in guiding their 
programme evaluation activities. Recent published 
examples of the application of the programme 
logic model are in fields of community rehabilita- 
tion (Letts 8t Dunal 1995), vocational rehabilita- 
tion (Lai & Mercier 2002), and case management 
(Aubry et al. 2000). It was also used as the basis for 
a workbook on programme evaluation for occupa- 
tional therapists in Canada (Letts et al. 1999). 
Some services have used the programme logic 
model to assist in the development of new pro- 
grammes, to be ready for evaluation (Dwyer 8t 
Makin 1997; Steinwender et al. 1998). 

A programme logic model is a chart that pro- 
vides basic information about a programme, its 
components and the linkages between what a pro- 
gramme does and what it is expected to achieve 
(Rush 8t Ogborne 1991). A key characteristic of 
this model is that the ‘means’ (what you do) and 
the ‘ends’ (the results or outcome of what you do) 
are separated. ‘Means’ refers to the process of the 
programme, and ‘ends’ are the outcomes (Letts 
et al. 1999). These two important aspects of a pro- 
gramme are clearly distinguished for an evaluation. 
There are five main elements in a programme logic 
model. The primary activities of the programme 
are identified as the Main Components. The Imple- 
mentation Objectives then translate the main com- 
ponents into objectives about what the programme 
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is expected to do. Outputs are the indicators or 
process components of the programme’s imple- 
mentation, and are comprised of the data about 
service delivery and the people who are served. 
Short-term and Long-term Outcome Objectives 
refer to what you expect to change or occur as a 
result of the programme over shorter or longer- 
time frames. Finally, the arrows link the various 
elements of the model to show the main ‘logic’ of 
the programme. 

The programme: transition services at a 
children’s rehabilitation centre 

A pilot programme at an outpatient children’s 
rehabilitation centre in the province of Ontario, 
Canada was implemented in 1999, with funding 
from the provincial Ministry of Health. The pro- 
gramme (called ‘transition services’) focused on 
providing extra therapy services to children in 
transition from pre-school to the elementary 
school system. Additional services covered under 
this new programme initiative were occupational 
therapy and physiotherapy, both at a 0.5 full- 
time equivalent (FTE) level. In addition to the 
extra therapy services provided individually to 
children and their families, Centre staff partici- 
pated in four evening parent education sessions. 
Parents who attended the education sessions 
received a binder with written information and 
resources. 

The evaluation 

Part of the funding for this new initiative was 
directed towards an evaluation, which was con- 
ducted by CanChild Centre for Childhood Disabil- 
ity Research, at McMaster University. CanChild is 
a Ministry of Health funded research unit with 
extensive experience in design and evaluation. It is 
formally partnered with children’s rehabilitation 
centres across Ontario, and this close relationship 
prompted the staff at the Children’s Centre to 
approach and work with CanChild. 

The programme evaluation began with prelimi- 
nary meetings to discuss the purpose and out- 
comes of the programme evaluation, and to clearly 
identify the people involved and their roles and 


responsibilities. This early work ensured a smooth 
process during the 10 months of evaluation. The 
budget for the evaluation was negotiated, and a 
research co-ordinator was hired part-time. An 
evaluation committee at CanChild was formed to 
oversee the process and provide input to data 
collection, analysis and interpretation. Ethics 
approval for the study was received from the uni- 
versity’s research ethics committee. 

Evaluation methods 

The design of the programme evaluation was 
driven by a programme logic model, which was 
developed collaboratively by staff at the Children’s 
Centre and CanChild (Fig. 1). A combination of 
quantitative and qualitative methods was used to 
measure the Outputs and Short-term objectives. The 
administration of the Children’s Centre collected 
service statistics as part of their regular operations, 
and this data was used to measure most of the 
Outputs of the programme. 

The measures used by the evaluation team were 
administered after the children had made the 
transition to kindergarten. They included the 
following: 

1 A chart audit tool, developed specifically for this 
evaluation, was used to collect data about the 
process of service delivery. Occupational therapy 
and physiotherapy students were hired and 
trained to conduct the chart audits. Inter-rater 
reliability of 0.70 or greater overall was estab- 
lished before chart abstraction. The chart audit 
tool was a checklist that allowed the extractors 
to record the number of occurrences of different 
types of service each child received. 

2 The Measure of Processes of Care (MPOC), 20- 
item version (King et al. 1998) is a questionnaire 
about parents’ perceptions of the services they 
and their children received. It uses a seven-point 
Likert rating scale. 

3 The Client Satisfaction Questionnaire (CSQ), 
adapted from Larsen et al. (1979), is completed 
by parents about their satisfaction with the ser- 
vices they and their child received. It contains 
nine questions about services using a four-point 
scale, plus two open-ended questions. 
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Main 

components 

Enabling Parents 

Intervention 

Enabling transition process 

Implementation 

objectives 

• To provide resource sessions 
and materials which assist 
parents, with skills to help 
meet the needs of their 
children in community schools. 

• To provide intervention as 
needed by the clients and 
their families as determined 
the goal setting process by 
increasing resources. Focus 
on school readiness skills. 

• To provide a consistent team for 
the family and client, while 
working together with the school 
in supporting transition. 

Outputs 

• Number of Information 
packages and materials to 
support knowledge, 
self-esteem/self-confidence, 
effective advocacy. 

• Number of education sessions 
provided. 

• Number of parents participating. 

• Level of OT and PT 
resources increased by 0.5 
FTE each from current 
levels. 

• Ratios of direct and indirect 
service levels to clients and 
families recorded 

• Number and type of contacts 
with schools. 

• Number and type of supports 
and resources for clients and 
families as they move into 
school. 

Short-term 

outcome 

objectives 

• Transition process is less 
stressful for parents. 

• Parents perceive they have 
adequate information to 
successfully address issues as 
they arise. 

• To support ongoing 
development of skills for 
transition. 

• Parents perceive that continuity 
and coordination of services are 
provided. 

• Parents are satisfied with 
resources put into place to 
support transition. 

Long-term 

outcome 

objectives 

To improve the transition process of clients and families, from preschool to school entry. 


Figure 1 . Programme logic model for transition services. FTE, full-time equivalent; OT, occupational therapy; PT, 
physiotherapy. 


4 Goal Attainment Scaling (GAS) (Kiresuk et al. 
1994) involves a therapist who is working with 
each child to set a goal in consultation with par- 
ents, and then identify five possible outcomes 
for the goal. The five levels include a baseline 
level (what the child is doing now), an expected 
or hoped-for outcome level, and outcomes that 
are above expectations. See Fig. 2 for an example 
of a GAS form completed for this evaluation. All 
goals were evaluated by an independent 
researcher to ensure that they were measurable, 
achievable and relevant. At the end of the 
evaluation period, an independent evaluator 
observed the child or interviewed the parent(s) 
to determine the level achieved for each goal. 

5 Qualitative interviews were conducted with par- 
ents and service providers who were purpose- 
fully selected by the research team to represent 
a broad range of experiences and perceptions of 
the services they and their child received. Qual- 
itative data were gathered to supplement the 
quantitative data, and to explore parents’ and 
service providers’ perceptions of the new ser- 
vices in greater depth. The evaluators were 
interested in exploring the factors that made a 
difference for some families - what they liked, 
what they didn’t like - and their recommenda- 
tions for the future. Interviews were tape- 
recorded and transcribed to text for analysis. 
The research team read the transcripts and 


developed a coding scheme of the key themes 
that emerged. One researcher used the coding 
scheme to analyse all transcripts after coding 
agreement of 75% was achieved. 

Sample selection 

The parents of all clients involved in the new tran- 
sition services during the evaluation period (1 Jan- 
uary 2000 to 31 October 2000) were approached to 
participate in this study. Parents were recruited by 
phone for their verbal consent to participate before 
providing informed written consent. Parents pro- 
vided demographic information about their child, 
their family and the services they were receiving. 

Results 

Sample demographics 

All clients born in 1995 or 1996 were eligible to 
start junior or senior kindergarten in the public 
school system in the year 2000/01. There were 81 
clients in the Centre’s catchment region who were 
eligible, and only 21 clients chose to start junior or 
senior kindergarten in this school year. Reasons 
parents chose to hold their children back from 
starting school included a lack of support for their 
child in kindergarten or parents feeling their child 
was not ready for the community school system. 
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Sample 

GOAL ATTAINMENT SCALING FORM 
for Transition Services Programme Evaluation 

Child’s name: James A. 

Therapy Discipline: Occupational Therapy 

• target area: productivity 

• sub-category: school readiness 

Therapist’s name: B.B. 

Goal Attainment Rating Scale: 

-2 James draws a vertical line by imitating an adult, to identify his paper work at 
school. 

-1 James draws a vertical line from memory (no imitation), to identify his paper 
work at school. 

0 James draws a recognizable ‘A’ from memory, to identify his paper work at 
school. 

+ 1 James draws a recognizable ‘JA’ from memory, to identify his paper work at 
school. 

+2 James prints a recognizable ‘JAMES’ from memory, to identify his paper work 
at school. 

Figure 2. Sample goal attainment scaling form for transition services programme evaluation. 


Sixteen clients consented to participate, how- 
ever, one client dropped out from the study for 
medical reasons, and two families did not return 
their questionnaires. Thirteen clients were 
included in the final analysis. The mean age of the 
13 children involved in the evaluation of the tran- 
sition services was 5.0 years (range 3.10-6.4 years). 
The primary diagnoses were: cerebral palsy 
(37.5%), spina bifida (12.5%), developmental 
delay (6.3%), syndromes (6.3%) and ‘other’ 
(18.8%), such as behavioural or communication 
disorders. 

The results of this evaluation are reported using 
the structure of the programme logic model. 


Outputs 

The service statistics collected by the administra- 
tion of the Children’s Centre demonstrated that 
additional occupational therapy and physiotherapy 
services were provided over the course of the 
10 months of the new initiative. The results of the 
chart audit indicated that 90% of the time, thera- 
pists used non-standardized assessment methods 
when assessing clients. These methods included 
clinical observation, interview with parents and 
others, and checklists developed by the therapists 
themselves. For intervention, therapists spent 
approximately 45% of their time on direct services 
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with the child, 30% in consultation, education and 
mediator training, and 17% in conferences and 
meetings (8% of time was not clearly documented 
in charts). 

Short-term objectives 

Quantitative results 

Goal attainment Goal Attainment Scaling was 
used to evaluate the achievement of goals set for 
each of the 13 children in the study. At least one 
goal was set for each child, and a rating scale was 
established with five levels (-2 to +2) representing 
five possible outcomes for each goal (see Fig. 2 for 
sample GAS). Raw scores were converted to T- 
scores to account for some clients having more 
than one GAS goal. A T- score of 30 equals the 
baseline score of -2; 40 equals a score of — 1 which 
is considered slightly less goal achievement than 
expected; 50 equals a level of 0 or expected out- 
come; 60 equals a score of +1, which is consider 
somewhat better than expected; and 70 equals +2 
or much better than expected). 

The mean T-score was for the transition services 
was 52.64, indicating achievement of a functional 
outcome slightly above what was expected by the 
therapists and parents who set the goals for each 
child. 

Parents’ perceptions of services The MPOC was 
used to determine the parents’ perceptions of the 
services they and their child received. Five subdo- 
main scores are calculated for the MPOC: enabling 
and partnership, providing general information, 
providing specific information, co-ordinated and 
comprehensive care, and respectful and supportive 
care. Thirteen parents (one for each child) com- 
pleted the MPOC questionnaire at the end of the 
evaluation period. 

The MPOC scores were compared to data from 
the Family Centred Service (FCS-II) study con- 
ducted in the province of Ontario in 1999 ( Can- 
child Centre for Childhood Disability Research 
1999). This study surveyed 494 parents from across 
Ontario. Comparisons are graphically represented 
in Fig. 3. This comparative analysis demonstrates 
that the MPOC scores for the transition services 


Transition programme 



Figure 3. Measure of Processes of Care (MPOC) scores - 
comparison between transition services and Family Centred 
Service (FCS-II) (1999) study provincial data. (Error bars 
indicate 95% confidence interval around mean), ep, 
enabling and partnership; pgi, providing general informa- 
tion; psi, providing specific information about the child; ccc, 
co-ordinated and comprehensive care for the child and fam- 
ily; rsc, respectful and supportive care. 


are significantly higher in all five domains when 
compared to the data from the FCS-II study. 

Client satisfaction: general Thirteen parents also 
completed the CSQ at the end of the evaluation 
period. Nine items use a four-point rating scale, 
with a score of 4 representing a high level of satis- 
faction (total possible score is 36). The mean raw 
score for the transition services was 30.72, with a 
range from 27.5 to 32.0. This score was compared 
to CSQ data from the FCS-II (1999) study in which 
the mean was 25.89. This demonstrates a higher- 
than-average level of parent satisfaction for this 
service compared to the Ontario-wide study (see 
Table 1). 

Qualitative results 

Qualitative data were gathered from parents and 
service providers about the factors that made a dif- 
ference, what they liked and didn’t like about the 
new services, and their ideas for improvement. 
Results are reported under the themes that 
emerged from the analysis. 
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Table 1 . Client Satisfaction Questionnaire (CSQ) scores for 
transition services 




95% Confidence 


Mean CSQ 

interval 



score 

Lower 

Upper 

Transition services 

30.72 

29.43 

32.01 

FCS-II (1999) study 

25.89 

25.45 

26.33 


Access to services Parents reported that location of 
the programme and therapists’ flexibility in sched- 
uling appointments promoted access to services. 
Barriers to access included busy schedules, and lack 
of babysitting services. Some parents reported a 
desire for increased access to specialists and 
equipment. 

Service co-ordination Parents reported satisfac- 
tion with the continuity and co-ordination 
between their therapists and external agencies, 
including the schools. In this newly established ser- 
vice, parents and therapists found the joint visits 
with the ‘old’ and ‘new’ therapists in the public 
schools, together with the parents and teachers, 
very helpful for co-ordination. 

Communication Parents were typically pleased 
with their ability to communicate with therapists, 
and openly express their concerns. They valued the 
personal relationships they had with their 
therapists. 

Progress and growth Parents consistently com- 
mented on how they felt the services had improved 
their child’s abilities in everyday activities, in par- 
ticular, their preparation for kindergarten. Their 
children also developed increased confidence, ini- 
tiative and independence as a result of the services. 

Family participation Parents frequently cited 
their appreciation of being involved in therapy ses- 
sions, and being given suggestions for activities to 
do at home. They identified their roles as experts, 
advocates and active team members, as well as their 
interest in defining the types of service delivery 
models they want their child to receive. 

Empowerment Parents suggested that being edu- 
cated and informed about their child’s needs and 


services allowed them to be more effective parents. 
Some parents who felt they lacked knowledge in 
this area, particularly as they started the transition 
services, expressed feelings of being lost, afraid and 
uncertain. Parents felt that the new services pro- 
vided them with insights about their child and 
about the school system that they would not have 
received otherwise. 

Change as the transition services were new, this 
resulted in changes for the families, in addition to 
the added stress of a major life change for their 
child entering the publicly funded school system. 
Parents recognized that change was always difficult, 
but they felt for the most part that the Centre staff 
were trying to be flexible and accommodating. Ser- 
vice providers reported that they also felt the stress 
of change, but felt that the earlier transition to the 
school-based team was worthwhile as it helped to 
ease some of the anxieties associated with starting 
school for some of the families. 

These themes highlight that parents felt the new 
therapy services were valuable in helping their chil- 
dren develop and grow, and were delivered using a 
family-centred approach. Communication and 
information helped parents to feel knowledgeable 
about their child’s needs and the services available. 
The parents offered suggestions for future services, 
including more information about their child’s 
development, and other possible services available 
within and outside of the Children’s Centre. 

Discussion 

This evaluation of a new service at an outpatient 
children’s rehabilitation centre provided valuable 
information about the process and outcomes of the 
services. The programme logic model served as a 
useful framework to guide the evaluation, and to 
meet the needs of both evaluators and Centre staff. 
The Outputs, or process of service delivery were 
evaluated through service statistics and a chart 
audit tool. Short-term objectives, or outcomes of the 
services were evaluated using a combination of 
quantitative and qualitative methods. 

The Outputs of this new service indicated that 
additional therapy services were provided to the 
study participants during the 10-month evaluation 
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period, as planned. The audits of therapy charts for 
the children involved in this evaluation revealed 
that assessments used primarily non-standardized 
methods such as observations, interviews and 
checklists. These methods provided therapists with 
qualitative information about the children they 
were working with, and helped them to plan 
intervention, however, non-standardized measures 
make it difficult to measure change in children over 
time. Intervention approaches were varied, with 
both direct service (with the child present) and 
indirect service (consultation, mediator training, 
etc.). This indicates that the therapists selected and 
adapted their service approaches to meet their 
clients’ needs. The nature and frequency of services 
identified through the chart audit were similar to 
other services at the children’s rehabilitation cen- 
tre, except for the higher percentage of conferences 
and meetings. Given the nature of this new service, 
it was not surprising to find that 17% of the ther- 
apists’ time involved conferences and meetings to 
ensure a smooth transition to a new team and 
environment. 

The Short-term Objectives, or outcomes, of the 
new transition services were measured by GAS, 
the MPOC the CSQ and qualitative interviews. The 
mix of quantitative and qualitative measures 
ensured that breadth and depth of outcomes were 
evaluated. 

The results of the GAS overall indicate that the 
children involved in this evaluation met or 
exceeded the expected level of goal attainment for 
the study period. The GAS scores for the transition 
services were close to the expected mean outcome 
of 50, which indicates that therapists in the transi- 
tion services set goals that were realistic and achiev- 
able for the time period. 

The MPOC and the CSQ results indicated that 
parents’ overall satisfaction with services was 
higher than the provincial average. On the MPOC, 
the scores in all five domains of parents’ percep- 
tions of services were significantly higher than the 
Ontario-wide study data. The small sample size 
means that these findings should be interpreted 
with some caution. Some of the higher scores, par- 
ticularly those related to information, may be 
partly because of the extra education sessions and 
binder of information parents received about tran- 


sition to school. The higher scores on both mea- 
sures could also be an indication that specific 
‘block’ periods of therapy with a focused purpose, 
such as the transition from one developmental 
stage to another for children, are viewed positively 
by parents. 

Parents’ and service providers’ perceptions of 
services were explored through qualitative inter- 
views. This information supported the findings 
from the standardized measures and question- 
naires. The parents who were interviewed provided 
many examples of strong service co-ordination, 
opportunities for partnership, and participation. 
They felt that their children were making progress 
as a result of these services. Information about 
their child and the services available was identified 
as a key theme that influenced the parents’ feelings 
of empowerment, and the parents who were inter- 
viewed suggested that more information is always 
welcome about their children’s development. Ser- 
vice providers also reported positive perceptions of 
the new service, and focused primarily on continu- 
ing to improve communication between the ‘old’ 
team that knew the family well, and ‘new’ team that 
started to work with the child and family within 
the public school system. 

Overall, it can be concluded that the Long-term 
Objective of this new Transition Service: ‘To 
improve the transition process of clients and fam- 
ilies, from pre-school to school entry’ was achieved 
for the clients and families who participated in this 
evaluation. 

The small number of participants limits gener- 
alizability of results to other settings, but the 
positive results support a continuation of this new 
service and ongoing programme evaluation using 
a programme logic model. Future evaluations 
could include a cost-benefit analysis of the new 
programme. 

Summary and conclusions 

This evaluation of a new service initiative at an 
outpatient children’s rehabilitation centre was 
guided by a programme logic model. Process and 
outcome elements for evaluation were clearly iden- 
tified in the model that was developed by Centre 
staff and researchers before the evaluation began. 
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A combination of quantitative and qualitative 
methods of evaluation was then used to measure 
the identified Outputs and Short-term objectives. 

The results of this evaluation suggest that the 
programme logic model was a useful way to orga- 
nize this evaluation. The model helped to guide the 
researchers and the Centre staff in using a system- 
atic approach to conducting a programme evalua- 
tion in a relatively short-time frame. The model 
promoted ongoing communication between the 
evaluation team and Centre staff, who met regu- 
larly to review it and make any adjustments. Feed- 
back from Centre staff at the end of the evaluation 
indicated that the programme logic model helped 
them to feel comfortable with and engaged in the 
programme evaluation. Based on the experience 
and results of this pilot study, this type of system- 
atic approach to programme evaluation appears to 
be a viable and useful option for other children’s 
services. 
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